Image capturing apparatus, image capturing sensor, and image capturing processing method

ABSTRACT

This invention is directed to reduce a deterioration in image quality after thinning operation when pixels are read out upon thinning. An image sensor is designed such that different gains are applied on a pixel basis. After periodical gains are applied to the respective pixels, the pixels of the same colors are added and averaged to obtain a low-pass filter effect before thinning processing for the pixels in the sensor, thereby preventing the occurrence of aliasing (moiré) due to thinning.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/376,808 filed Mar. 15, 2006, which claims the benefit of JapanesePatent Application No. 2005-073235 filed on Mar. 15, 2005, which arehereby incorporated by reference herein their entirety.

FIELD OF THE INVENTION

The present invention relates to an image capturing apparatus, imagesensor, and image capturing processing method and, more particularly, toa technique suitably used for adding/thinning processing for pixels.

BACKGROUND OF THE INVENTION

Conventionally, there has been available a moving image reading mode(see the CCD reading method (the horizontal & vertical pixel additiontechnique (corresponding to VGA moving images)); see Internet<URL:http://www.sony.co.jp/Products/SC-HP/imagingdevice/ccd/tecvga.html>) of reading out pixels from an image sensor at high speed bypixel adding/thinning operation, in addition to a frame reading mode ofsequentially reading out all pixels from the image sensor for stillimage photography. FIG. 11 is a view for explaining a conventionalmethod of reading out signals from a general-purpose CCD image sensor inthe moving image photography mode.

Referring to FIG. 11, after pixels are exposed, two pixels of the samecolor are added at a time in a vertical shift register. For example,pixels corresponding to numbers V in the vertical direction whichsatisfy V=10n (where n is 0 or a positive integer) and V=10n+2 areadded, and pixels which satisfy V=10n+5 and V=10n+7 are added. Theremaining pixels are not read out. Image signals added by a verticalshift register are transferred to a horizontal shift register.

In the horizontal shift register, pixels of the same color correspondingto numbers H in the horizontal direction which satisfy H=4m (m is 0 or apositive integer) and H=4m+2 are added, and pixels of the same colorwhich satisfy H=4m+1 and H=4m+3 are added. The resultant pixel data areoutput.

Such an image sensor outputs pixel data such that the number of pixelsin the vertical direction is reduced to ⅕, and the number of pixels inthe horizontal direction is reduced to ½.

In adding processing performed in the above manner, for example, asshown in FIG. 12, when G1[n+1] and G1[n+3] are added to output G1[K+1]and R[n] and R[n+2] are added to output R[K], a pixel coordinateinversion phenomenon occurs at G1[n+1] and R[n+2]. This results in theoccurrence of jaggies (a stair-step pattern) on a fine object image.

More specifically, when pixels having undergone the above addingprocessing are thinned and reduced to be output as, for example, aVGA-size image, jaggies are made less noticeable. In contrast to this,as the size of an output image increases, the influence of jaggiesincreases. If, therefore, the size of an output image increases, a greatdeterioration in image quality may occur.

As described above, according to an addition reading method in aconventional image capturing apparatus, when, for example, a G1-R lineis taken into consideration, the positional relationship between G1[n+1]and R[n+2] is reversed, as shown in FIG. 12. As a consequence, jaggiesappear on an oblique line or the like, resulting in a greatdeterioration in image quality.

In general, in sampling an image, if the signal band of an originalimage is not reduced to a frequency (Nyquist frequency) ½ the samplingfrequency, an aliasing signal is produced in a low-frequency signal.According to the addition reading method shown in FIGS. 11 and 12, pixeldata are read out while the number of pixels in the horizontal directionis reduced to ½, and the number of pixels in the vertical direction isreduced to ⅕.

However, the above addition of pixels alone cannot sufficiently suppressthe band of an original image. As a result, an aliasing signal isproduced in a low-frequency signal.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the aboveproblems and has as its first object to reduce a deterioration in imagequality after thinning operation when pixels are read out upon thinning.

It is a second object of the present invention to reduce aliasing whichoccurs after thinning operation.

It is a third object of the present invention to reduce jaggies whichare produced after thinning operation.

According to a first aspect of the present invention, there is providedan image capturing apparatus comprising signal amplifying means foramplifying pixel signals with amplification factors having a regularpattern set for a plurality of pixels arranged in the form of a matrix,pixel mixing means for mixing the pixel signals of the same colorsamplified by the signal amplifying means, and pixel thinning means forthinning and outputting the pixel signals mixed by the pixel mixingmeans.

According to a second aspect of the present invention, there is providedan image sensor comprising a plurality of pixels arranged in the form ofa matrix, signal amplification means for amplifying pixel signals withamplification factors having a regular pattern set for the plurality ofpixels, pixel mixing means for mixing output signals from thephotoelectric conversion unit for each pixel of the same color, andpixel thinning means for thinning pixel signals mixed by the pixelmixing means with a predetermined thinning ratio.

According to a third aspect of the present invention, there is providedan image capturing processing method comprising a signal amplifying stepof amplifying pixel signals with amplification factors having a regularpattern set for a plurality of pixels arranged in the form of a matrix,a pixel mixing step of mixing the pixel signals of the same colorsamplified in the signal amplifying step, and a pixel thinning step ofthinning and outputting the pixel signals mixed in the pixel adding andaveraging step.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the hardware arrangementof an image capturing apparatus according to the first embodiment of thepresent invention;

FIG. 2 is a view conceptually showing an example of a reading method inan addition mode according to the first embodiment of the presentinvention;

FIG. 3 is a circuit diagram showing an example of the pixel structure ofan image sensor according to the first embodiment of the presentinvention;

FIGS. 4A and 4B are views conceptually showing how adding and averagingoperation is performed in the image sensor according to the firstembodiment of the present invention;

FIG. 5 is a circuit diagram showing an example of the schematicarrangement of a G1 shift register in an RG horizontal shift registeraccording to the first embodiment of the present invention;

FIG. 6 is a circuit diagram showing the first example of the schematicarrangement of an R shift register in the RG horizontal shift registeraccording to the first embodiment of the present invention;

FIG. 7 is a circuit diagram showing the second example of the schematicarrangement of the R shift register in the RG horizontal shift registeraccording to the first embodiment of the present invention;

FIG. 8 is a view conceptually showing an example of a reading method inan addition mode according to the second embodiment of the presentinvention;

FIG. 9 is a view conceptually showing an example of a method of readingout pixels upon thinning of the pixels to ⅓ in the horizontal directionin the addition mode according to the third embodiment of the presentinvention;

FIG. 10 is a view conceptually showing an example of a method of readingout pixels upon thinning of the pixels to ⅕ in the horizontal directionin the addition mode according to the third embodiment of the presentinvention;

FIG. 11 is a view for explaining a conventional method of reading outsignals from a general-purpose CCD image sensor in a moving imagephotography mode according to the prior art; and

FIG. 12 is a view conceptually showing how adding and averagingoperation is performed in an image sensor according to the prior art.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of the present invention will be described next withreference to the accompanying drawings.

First Embodiment

An example of the hardware arrangement of an image capturing apparatusaccording to this embodiment will be described first with reference toFIG. 1.

Reference numeral 101 denotes an optical system comprising a lens and astop; 102, a mechanical shutter; and 103, an image sensor. The imagesensor 103 in this embodiment includes a color filter having a primarycolor Bayer pattern. Reference numeral 104 denotes a CDS circuit whichperforms analog signal processing; and 105, an A/D converter whichconverts an analog signal into a digital signal.

Reference numeral 106 denotes a timing signal generating circuit whichgenerates signals for operating the image sensor 103, the CDS circuit104, and the A/D converter 105; 107, a driving circuit for the opticalsystem 101, mechanical shutter 102, and image sensor 103; 108, a signalprocessing circuit which performs signal processing necessary forcaptured image data; and 109, an image memory which stores image datahaving undergone signal processing.

Reference numeral 110 denotes an image recording medium detachable withrespect to the image capturing apparatus; 111, a recording circuit whichrecords image data having undergone signal processing on the imagerecording medium 110; 112, an image display apparatus which displaysimage data having undergone signal processing; and 113, a displaycircuit which causes the image display apparatus 112 to display animage.

Reference numeral 114 denotes a system control unit which controls theoverall image capturing apparatus; and 115, a non-volatile memory (ROM).The non-volatile memory (ROM) 115 stores, for example, a programdescribing a control method executed by the system control unit 114,control data such as parameters and tables which are used when theprogram is executed, and correction data for flaw addresses and thelike. Reference numeral 116 denotes a volatile memory (RAM). Theprogram, control data, and correction data stored in the non-volatilememory 115 are transferred and stored in the volatile memory (RAM) 116in advance to allow the system control unit 114 to use them whencontrolling the image capturing apparatus.

An example of the image capturing operation performed by the imagecapturing apparatus having the above arrangement using the mechanicalshutter 102 will be described next.

Assume that before image capturing operation, at the start of operationof the system control unit 114, e.g., at power-on of the image capturingapparatus, the system control unit 114 transfers necessary programs,control data, and correction data from the non-volatile memory 115 tothe volatile memory 116 and stores them therein in advance.

Assume also that such programs and data are used by the system controlunit 114 to control the image capturing apparatus, and additionalprograms and data are transferred from the non-volatile memory 115 tothe volatile memory 116 as needed, or the system control unit 114directly reads out data from the non-volatile memory 115 and uses them.

First of all, the optical system 101 drives the stop and the lens inaccordance with control signals from the system control unit 114 to forman object image on the image sensor 103 with properly set brightness.The mechanical shutter 102 is then driven by a control signal from thesystem control unit 114 to shield the image sensor 103 from light inaccordance with the operation of the image sensor 103 so as to set anecessary exposure time. If the image sensor 103 has an electronicshutter function, the function may be used together with the mechanicalshutter 102 to ensure a necessary exposure time.

The image sensor 103 is driven by a driving pulse based on an operationpulse generated by the timing signal generating circuit 106 controlledby the system control unit 114. The image sensor 103 converts an objectimage into an electrical signal by photoelectric conversion and outputsit as an analog image signal. The analog image signal output from theimage sensor 103 is converted into a digital image signal by the A/Dconverter 105 after clock synchronous noise is removed from the signalby the CDS circuit 104 in accordance with an operation pulse generatedby the timing signal generating circuit 106 controlled by the systemcontrol unit 114.

The signal processing circuit 108 controlled by the system control unit114 then performs image processing such as color conversion, whitebalance, and gamma correction, resolution conversion processing, imagecompression processing, and the like for the digital image signal. Theimage memory 109 is used to temporarily store a digital image signalduring signal processing or store image data as a digital image signalhaving undergone signal processing. The image data having undergonesignal processing in the signal processing circuit 108 or the image datastored in the image memory 109 is converted into data (e.g., file systemdata having a hierarchical structure) suitable for the image recordingmedium 110 by the recording circuit 111 and recorded on the imagerecording medium 110. Alternatively, this signal is subjected toresolution conversion processing in the signal processing circuit 108,and the resultant data is converted into a signal (e.g., an NTSC analogsignal) suitable for the image display apparatus 112 by the displaycircuit 113 and displayed on the image display apparatus 112.

In this case, the signal processing circuit 108 may output a digitalimage signal as image data to the image memory 109 or the recordingcircuit 111 without performing signal processing in accordance with acontrol signal from the system control unit 114. Upon receiving arequest from the system control unit 114, the signal processing circuit108 outputs the information of a digital image signal or image datagenerated in the process of signal processing, e.g., the information ofthe spatial frequency of an image, the average value of a designatedregion, and the data amount of a compressed image, or informationextracted therefrom to the system control unit 114. In addition, uponreceiving a request from the system control unit 114, the recordingcircuit 111 outputs information, e.g., the type and remaining capacityof the image recording medium 110, to the system control unit 114.

FIG. 2 is a view conceptually showing a reading method in the additionmode in the image capturing apparatus according to this embodimenthaving the above arrangement. FIG. 3 is a view showing an example of thepixel structure of the image sensor according to this embodiment. Notethat FIG. 2 shows an example in which the image sensor has a pluralityof pixels arranged in the form of a 6 (row)×11 (column) matrix.Referring to FIG. 2, R, G1, G2, and B represent the colors of pixels (Rrepresenting red; G1 and G2, green; and B, blue), and “x1” and “x2”shown under R, G1, G2, and B represent gains (amplification factors)applied to the respective pixels. The main operation of the imagecapturing apparatus according to this embodiment will be described belowwith reference to FIGS. 2 and 3.

(Step 1-1)

First of all, an object image passing through the optical system 101upon exposure operation is formed on the image sensor 103 shown in FIG.2. As shown in FIG. 3, a photodiode 203 placed in each pixel of theimage sensor 103 outputs an electrical signal (current) corresponding toa light amount.

As compared with a conventional image sensor, the image sensor 103 ofthis embodiment is mainly characterized by including amplifiers (anormal mode amplifier 201 and an addition mode amplifier 202) capable ofapplying different gains for each pixel. In reading operation in thenormal mode, transistors 206 and 208 are turned on by a still imageselect (s select) signal 204, and an electrical signal output from thephotodiode 203 is amplified by the normal mode amplifier 201 on thecapacitor Cs side, thereby obtaining a pixel output signal.

In reading operation in the addition mode, transistors 207 and 209 areturned on by an addition select (m select) signal 205, an electricalsignal is amplified by the addition mode amplifier 202 on the capacitorCm side, thereby obtaining a pixel output signal.

Note that each pixel is reset by using reset MOS transistors 210 and 211which are turned on by a reset signal 212.

(Step 1-2)

Referring to FIG. 2, on an R-G1 line (a line whose number V in thevertical direction is an odd number, V=2m+1 (where m is 0 or a positiveinteger)), the addition mode amplifier 202 applies a x2 gain to an Rpixel at a position where a number H in the horizontal directionsatisfies H=4n (where n is 0 or a positive integer).

On a G2-B line (a line whose number V in the vertical direction is aneven number, V=2m (where m is 0 or a positive integer)), the additionmode amplifier 202 applies a x2 gain to a B pixel at a position where anumber H in the horizontal direction satisfies H=4n+3 (where n is 0 or apositive integer). Assume that the gain amount at each of the remainingpixels is x1.

(Step 1-3)

Vertical transfer is started.

(Step 1-4)

In an RG horizontal shift register 103 a and GB horizontal shiftregister 103 b, pixels of the same colors are added. At this time, pixelvalues from even-numbered and odd-numbered lines in the verticaldirection are transferred to different horizontal shift registers (theRG horizontal shift register 103 a and the GB horizontal shift register103 b), and different pixel adding operations are executed in therespective registers. The resultant data are then transferred in thehorizontal direction.

An example of pixel adding operation on the R-G1 (a line whose number Vin the vertical direction in FIG. 2 is an odd number) will be describedbelow.

Two adjacent pixels G1 of the same color are added and averaged, and theresultant pixel is transferred to the RG horizontal shift register 103a. FIGS. 4A and 4B are views conceptually showing how adding andaveraging is performed by the image sensor 103 in this embodiment. FIG.4A shows that a pixel G1[n+1] and a pixel G1[n+3] are added and averagedto generate a pixel G1[K+1].

FIG. 5 is a view showing an example of the schematic arrangement of a G1shift register in the RG horizontal shift register 103 a.

Referring to FIG. 5, in the normal reading mode, each pixel G1[n+1] issequentially output by a still image select (s select) signal 401 and astill image serial register 403 through transistors 405 and 406. In thenormal reading mode, each pixel G1[n+3] is sequentially output by astill image select signal 401 and the still image serial register 403through transistors 407 and 408.

In the addition reading mode, the pixels G1[n+1] and G1[n+3] are addedand averaged by an addition select (m select) signal 402 throughtransistors 409 and 410 in a G addition unit (adding and averagingcircuit GAdd[k+1]) 412, and output by an addition serial register 404through transistors 411. Likewise, the pixel G1[n+5] and the pixelG1[n+7] are added and averaged by a G addition unit (adding andaveraging circuit GAdd[k+3]) 413. The above operation can be expressed,for example, by

$\begin{matrix}{{{G\;{1\lbrack {K + 1} \rbrack}} = {( {{G\;{1\lbrack {n + 1} \rbrack}} + {G\;{1\lbrack {n + 3} \rbrack}}} )/2}}{{G\;{1\lbrack {K + 3} \rbrack}} = {( {{G\;{1\lbrack {n + 5} \rbrack}} + {G\;{1\lbrack {n + 7} \rbrack}}} )/2}}\mspace{79mu}\vdots} & (1)\end{matrix}$

Subsequently, three adjacent pixels R of the same color are added andaveraged, and the resultant pixel is transferred to the RG horizontalshift register 103 a. FIG. 4A shows that pixels R[n+2], R[n+4], andR[n+6] are added and averaged to generate a pixel R[k+2].

FIG. 6 is a view showing an example of the schematic arrangement of an Rshift register in the RG horizontal shift register 103 a.

Referring to FIG. 6, in the normal reading mode, each pixel R[n+2] issequentially output by a still image select (s select) signal 501 and astill image serial register 503 through transistors 505 and 506.

In the normal reading mode, each pixel R [n+4] is sequentially output bya still image select (s select) signal 501 and a still image serialregister 503 through transistors 507 and 508.

In the normal reading mode, each pixel R[n+6] is sequentially output bythe still image select (s select) signal 501 and the still image serialregister 503 through transistors 509 and 510.

In the addition reading mode, the pixels R[n+2], R[n+4], and R[n+6] areadded and averaged by an addition select (m select) signal 502 and anaddition serial register 504 through transistors 511, 512, and 513 in anR addition unit (adding and averaging circuit RAdd[k+2]) 514. Likewise,the pixels R[n+6], R[n+8], and R[n+10] are added and averaged by an Raddition unit (adding and averaging circuit RAdd[K+4]) 515.

The above operation can be expressed, for example, by

$\begin{matrix}{{{R\lbrack {K + 2} \rbrack} = {( {{\alpha \times {R\lbrack {n + 2} \rbrack}} + {\beta \times {R\lbrack {n + 4} \rbrack}} + {\Upsilon \times {R\lbrack {n + 6} \rbrack}}} )/4}}{{R\lbrack {K + 4} \rbrack} = {( {{\alpha \times {R\lbrack {n + 6} \rbrack}} + {\beta \times {R\lbrack {n + 8} \rbrack}} + {\Upsilon \times {R\lbrack {n + 10} \rbrack}}} )/4}}\mspace{20mu}\vdots} & (2)\end{matrix}$

In this case, the coefficients α and γ are both “1” (α=γ=1), and thecoefficient β is “2” (β=2). These coefficients are multiplied on pixelsin the processing of (Step 1-2) described above.

The pixels obtained by the above adding and averaging operation andtransferred to the RG horizontal shift register 103 a are sequentiallytransferred and output from the RG horizontal shift register 103 a.

An example of pixel adding operation on the G2-B (a line whose number Vin the vertical direction in FIG. 2 is an even number) will be describedbelow. Note that since the same arrangement and operation as those forthe R-G1 line described with reference to FIGS. 5 and 6 are applied tothe G2-B line, adding operation will be briefly described, and adetailed description thereof will be omitted.

Two adjacent pixels G2 of the same color are added and averaged, and theresultant pixel is transferred to the GB horizontal shift register 103b. FIG. 4B is a view showing that pixels G2[n] and G2[n+2] are added andaveraged to generate a pixel G2[K]. This operation can be expressed, forexample, by

$\begin{matrix}{{{G\;{2\lbrack K\rbrack}} = {( {{G\;{2\lbrack n\rbrack}} + {G\;{2\lbrack {n + 2} \rbrack}}} )/2}}{{G\;{2\lbrack {K + 2} \rbrack}} = {( {{G\;{2\lbrack {n + 4} \rbrack}} + {G\;{2\lbrack {n + 6} \rbrack}}} )/2}}\mspace{20mu}\vdots} & (3)\end{matrix}$

Subsequently, three adjacent pixels B of the same color are added andaveraged, and the resultant pixel is transferred to the GB horizontalshift register 103 b. FIG. 4B shows that pixels B[n+1], B[n+3], andB[n+5] are added and averaged to generate a pixel B[k+1]. This operationcan be expressed, for example, by

$\begin{matrix}{{{B\lbrack {K + 1} \rbrack} = {( {{\alpha \times {B\lbrack {n + 1} \rbrack}} + {\beta \times {B\lbrack {n + 3} \rbrack}} + {\Upsilon \times {B\lbrack {n + 5} \rbrack}}} )/4}}{{B\lbrack {K + 3} \rbrack} = {( {{\alpha \times {B\lbrack {n + 5} \rbrack}} + {\beta \times {B\lbrack {n + 7} \rbrack}} + {\Upsilon \times {B\lbrack {n + 9} \rbrack}}} )/4}}\mspace{20mu}\vdots} & (4)\end{matrix}$

In this case, the coefficients α and γ are both “1” (α=γ=1), and thecoefficient a is “2” (β=2). These coefficients are multiplied on pixelsin the processing of (Step 1-2) described above.

The pixels obtained by the above adding and averaging operation andtransferred to the GB horizontal shift register 103 b are sequentiallytransferred and output from the GB horizontal shift register 103 b.

In this embodiment, as described above, different gains are applied foreach pixel unit in the image sensor 103, and pixels of the same colorsare added and averaged after a gain is periodically applied to eachpixel. This makes it possible to obtain a low-pass filter effect beforethinning processing is performed for the pixels in the sensor, therebypreventing the occurrence of aliasing (moiré) due to thinning.

In addition, R and B signals are added and averaged along the horizontaldirection three pixels at a time, and G signals are added and averagedalong the horizontal direction two pixels at a time. This can preventthe pixel position relationships between the G and R signals and betweenthe G and B signals from being reversed. This can greatly prevent theoccurrence of jaggies due to the reversal of pixel positions whichoccurs in the conventional addition reading mode.

In this embodiment, an amplifier is provided for each pixel to apply again for each pixel. However, as shown in FIG. 7, a vertical transferline may include an amplifier capable of applying a gain to each pixel.

Referring to FIG. 7, reading operation in the normal mode is performedby a still image select (s select) signal 601, and reading operation inthe addition mode is performed by an addition select (m select) signal602.

When reading operation in the addition mode is performed by the additionselect (m select) signal 602, a 1× gain is applied to the pixel R[n+2]by an amplifier 608 through a transistor 605. A 2× gain is applied tothe pixel R[n+4] by an amplifier 609 through a transistor 606. The pixelR[n+6] passes through a transistor 607, and an amplifier 610 applies a1× gain to the pixel.

The pixels R[n+2], R[n+4], and R[n+6] after gain correction are addedand averaged by an R addition unit (adding and averaging circuitRadd[K+2]) 611, and the resultant pixel is output as a pixel R[K+2]through a transistor 612. The arrangement shown in FIG. 7 requires aselector for applying different gains for the respective colors.However, there is no need to hold an amplifier for each pixel, and hencethe chip area can be reduced. Note that the same adding and averagingoperation as that described above is performed in an R addition unit(adding and averaging circuit Radd[K+4]) 613.

In this embodiment, adding processing and thinning processing areperformed within the image sensor 103. Obviously, however, an image withlittle aliasing can be obtained by reading out all pixels from the imagesensor 103 in the still image mode and performing adding/thinningprocessing by using a program in the image capturing apparatus or a PC(Personal Computer).

Second Embodiment

The second embodiment of the present invention will be described next.This embodiment performs adding processing in the sensor verticaldirection in addition to adding processing in the sensor horizontaldirection in the first embodiment, thereby preventing a deterioration inimage quality due to thinning processing in the vertical direction. Asdescribed above, this embodiment differs from the first embodiment inthat adding processing is performed in the sensor vertical direction,but other arrangements are the same. The same reference numerals inFIGS. 1 to 7 as in the first embodiment denote the same portions in thesecond embodiment, and a detailed description thereof will be omitted.

In an image sensor 103, circuit arrangements for adding processing inthe horizontal direction, of circuit arrangements for applying differentgains for each pixel and circuit arrangements for pixel addition, arethe same as those in the first embodiment. Since it is obvious thatadding processing in the vertical direction can be realized by makingthe circuit arrangement of a vertical shift register have the samecircuit arrangement as that of the horizontal shift register in thefirst embodiment, a detailed description thereof will be omitted (seeFIGS. 5 to 7).

FIG. 8 is a view conceptually showing an example of a reading method inthe addition mode in an image capturing apparatus according to thisembodiment. Note that FIG. 8 shows an example in which an image sensorhas a plurality of pixels arranged in the form of a six (row)×7 (column)matrix. Referring to FIG. 8, R, G1, G2, and B represent the colors ofpixels (R representing red; G1 and G2, green; and B, blue), and “x1” and“x2” shown under R, G1, G2, and B represent gains (amplificationfactors) applied to the respective pixels.

(Step 2-1)

First of all, an object image passing through an optical system uponexposure operation is formed on the image sensor 103. As in the firstembodiment, a photodiode 203 placed in each pixel of the image sensor103 then outputs an electrical signal (current) corresponding to a lightamount.

(Step 2-2)

On an R-G1 line (a line whose number V in the vertical direction is anodd number, V=2m+1 (where m is 0 or a positive integer)), an amplifierapplies a x2 gain to an electrical signal at an R pixel at a positionwhere a number H in the horizontal direction satisfies H=4n (where n is0 or a positive integer). On a G2-B line (a line whose number V in thevertical direction is an even number, V=2m (where m is 0 or a positiveinteger)), an amplifier applies a x2 gain to an electrical signal at a Bpixel at a position where a number H in the horizontal directionsatisfies H=4n+3 (where n is 0 or a positive integer). Assume that thegain amount at each of the remaining pixels is x1.

(Step 2-3)

In vertical transfer operation, pixels of the same colors are added.Referring to FIG. 8, pixels on a line whose number V in the verticaldirection satisfies V=6n (n is 0 or a positive integer) and pixels on aline which satisfies V=6n+2 (n is 0 or a positive integer) are added andaveraged in the vertical direction. In addition, pixels on a line whosenumber V in the vertical direction satisfies V=6n+3 (n is 0 or apositive integer) and pixels on a line which satisfies V=6n+5 (n is 0 ora positive integer) are added and averaged in the vertical direction.Furthermore, pixels on a line whose number V in the vertical directionsatisfies V=6n+1 (n is 0 or a positive integer) and pixels whichsatisfies V=6n+4 (n is 0 or a positive integer) are skipped.

(Step 2-4)

In an RG horizontal shift register 103 a and GB horizontal shiftregister 103 b, pixels of the same colors are added. At this time, pixelvalues from an even-numbered line (G2-B line) and an odd-numbered line(R-G1 line) in the vertical direction are transferred to differenthorizontal shift registers (the RG horizontal shift register 103 a andthe GB horizontal shift register 103 b), and different pixel addingoperations are executed in the respective registers. The resultant dataare then transferred in the horizontal direction. Since a method ofadding pixels in the horizontal shift registers in this embodiment isthe same as that in the first embodiment, a description thereof will beomitted.

As described above, according to this embodiment, even if adding andaveraging operation is performed in both the sensor horizontal directionand the sensor vertical direction, the positional relationship betweenthe pixels of G and R signals and between G and B signals can beprevented from being reversed. This can prevent the occurrence ofjaggies. In addition, since four pixels of the same color are added, animage signal with little noise can be output.

In this embodiment as well, adding processing and thinning processingare performed within the image sensor 103. Obviously, however, an imagewith little aliasing can be obtained by reading out all pixels from theimage sensor 103 in the still image mode and performing addingprocessing and thinning processing on a program in the image capturingapparatus or a PC (Personal Computer).

Third Embodiment

The third embodiment of the present invention will be described next.This embodiment is characterized in that after gains having a commonpattern are applied to pixels of each color first and the pixels of thesame colors are added and averaged in a horizontal shift register toobtain a low-pass filter effect, the pixels are thinned and read out.This embodiment is also characterized in that the period of a gainapplied to pixels of the same color is changed in accordance with athinning ratio. Other arrangements are the same as those in the firstand second embodiments. Therefore, the same reference numerals in FIGS.1 to 8 as in the first and second embodiments described above denote thesame portions in the third embodiment, and a detailed descriptionthereof will be omitted.

FIG. 9 is a view conceptually showing an example of a method of readingout pixels upon thinning them to ⅓ in the horizontal direction in theaddition mode in an image sensor according to this embodiment.

Referring to FIG. 9, R, G1, G2, and B represent the colors of pixels (Rrepresenting red; G1 and G2, green; and B, blue), and “x1” and “x2”shown under R, G1, G2, and B represent gains (amplification factors)applied to the respective pixels shown thereon.

(Step 3-1)

First of all, an object image passing through an optical system uponexposure operation is formed on an image sensor 103. A photodiode 203placed in each pixel of the image sensor 103 outputs an electricalsignal (current) corresponding to a light amount.

(Step 3-2)

As shown in FIG. 9, gains having a predetermined pattern set in advancefor pixels of each color are applied by amplification circuits(amplifiers) provided for the respective pixels like those described inthe first embodiment. Referring to FIG. 9, gains having a regularity of(x1, x2, x1) are applied in the horizontal direction.

(Step 3-3)

Vertical transfer is started.

(Step 3-4)

The horizontal shift register adds pixels of the same colors and thenoutputs the resultant pixels.

An example of pixel adding operation on a G-B line (a line whose numberk in the vertical direction in FIG. 9 is an odd number) will bedescribed.

More specifically, for example, addition is executed by equations (5)given below:G[2m]=(G[6n]+G[6n+2]+G[6n+4])/4B[2m+1]=(B[6n+1]+B[6n+3]+B[6n+5])/4  (5)

An example of pixel adding operation on an R-G line (line whose number kin the vertical direction in FIG. 9 is an even number) will be describednext.

As in the case of the G-B line, addition is executed by, for example,equation (6) given below:G[2m+1]=(G[6n+1]+G[6n+3]+G[6n+5])/4R[2m]=(R[6n]+R[6n+2]+R[6n+4])/4  (6)

After data are output from the sensor in this manner, the number ofpixels in the horizontal direction is reduced to ⅓. In order to performthinning in the vertical direction, the vertical shift register mayperform adding and averaging processing like that performed by thehorizontal shift register as in the second embodiment.

A case wherein a mode of reading out pixels upon thinning them to ⅕ inthe horizontal direction is set will be described next. FIG. 10 is aview conceptually showing an example of the method of reading out pixelsupon thinning them to ⅕ in the horizontal direction in the addition modein the image sensor in this embodiment. Referring to FIG. 10, R, G1, G2,and B represent the colors of pixels (R representing red; G1 and G2,green; and B, blue), and “x1”, “x4”, and “x6” shown under R, G1, G2, andB represent gains (amplification factors) applied to the respectivepixels. As is obvious from FIGS. 9 and 10, in the mode of thinningpixels to ⅓, regular gains to be applied to pixels of the same colors inthe horizontal direction have a three-pixel period. In contrast to this,in the mode of thinning pixels to ⅕, gains have a five-pixel period. Asdescribed above, in this embodiment, the period of the pattern of gainsis increased as the thinning ratio increases.

(Step 4-1)

An object image passing through the optical system upon exposureoperation is formed on the image sensor. A photodiode placed in eachpixel of the image sensor outputs an electrical signal (current)corresponding to a light amount.

(Step 4-2)

As shown in FIG. 10, gains having a predetermined pattern set in advancefor pixels of each color are applied by amplification circuits(amplifiers) provided for the respective pixels like those described inthe first embodiment. Referring to FIG. 10, gains having a regularity of(x1, x4, x6, x4, x1) are applied in the horizontal direction.

(Step 4-3)

Vertical transfer is started.

(Step 4-4)

The horizontal shift register adds pixels of the same colors and thenoutputs the resultant pixels.

An example of pixel adding operation on a G-B line (a line whose numberk in the vertical direction in FIG. 10 is an odd number) will bedescribed.

More specifically, for example, addition is executed by equations (7)given below:G[2m]=(G[10n]+G[10n+2]+G[10n+4]+G[10n+6]+G[10n+8])/16B[2m+1]=(B[10n+1]+B[10n+3]+B[10n+5]+B[10n+7]+B[10n+9])/16  (7)

An example of pixel adding operation on an R-G line (line whose number kin the vertical direction in FIG. 10 is an even number) will bedescribed next.

As in the case of the G-B line, addition is executed by, for example,equations (8) given below:G[2m+1]=(G[10n+1]+G[10n+3]+G[10n+5]+G[10n+7]+G[10n+9])/16R[2m]=(R[10n]+R[10n+2]+R[10n+4]+R[10n+6]+R[10n+8])/16  (8)

After data are output from the sensor in this manner, the number ofpixels in the horizontal direction is reduced to ⅕. In order to performthinning in the vertical direction, the vertical shift register mayperform adding and averaging processing like that performed by thehorizontal shift register as in the second embodiment.

As described above, in this embodiment, the period of periodic gains ischanged depending on a thinning ratio, aliasing due to adding/thinningoperation can be reduced more reliably in addition to the effects of thefirst and second embodiments.

In this embodiment as well, adding processing and thinning processingare performed within the image sensor 103. Obviously, however, an imagewith little aliasing can be obtained by reading out all pixels from theimage sensor 103 in the still image mode and performing addingprocessing and thinning processing on a program in the image capturingapparatus or a PC (Personal Computer).

According to the embodiment described above, after pixel signals areamplified by amplification factors having a regular pattern set for aplurality of pixels arranged in the form of a matrix, the pixel signalsof the same colors are thinned upon adding and averaging operation. Thismakes it possible to obtain a low-pass filter effect before thinningprocessing for the pixels, thereby preventing aliasing due to thinningas much as possible.

In addition, R signals on the R-G line are added and averaged threepixels at a time, and G signals are added and averaged two pixels at atime. On the other hand, B signals on the G-B line are added andaveraged three pixels at a time, and G signals are added and averagedtwo pixels at a time. This can prevent the pixel position relationshipsbetween the G and R signals and between the G and B signals from beingreversed as much as possible. This can prevent the occurrence of jaggiesas much as possible.

Other Embodiment of the Present Invention

The present invention also incorporates the following arrangement withinits category. In this arrangement, in order to make the respectivedevices to operate so as to realize the functions of the aboveembodiments, the program codes of software for realizing the functionsof the above embodiments are supplied to a computer in an apparatus orsystem connected to the respective devices, thereby causing the computer(CPU or MPU) in the system or apparatus to operate the respectivedevices in accordance with the program codes.

In this case, the program codes of the software themselves realize thefunctions of the above embodiments, and the program codes themselves anda means for supplying the program codes to the computer, i.e., arecording medium in which the program codes are stored, constitute thepresent invention. As a recording medium storing such program codes, forexample, a flexible disk, hard disk, optical disk, magnetooptical disk,CD-ROM, magnetic tape, non-volatile memory card, ROM, or the like can beused.

Obviously, the above program codes are included in the embodiments ofthe present invention not only in a case wherein the functions of theabove embodiments are realized when the computer executes the suppliedprogram codes but also in a case wherein the functions of the aboveembodiments are realized by the program codes in cooperation with the OS(Operating System), another application software, or the like running onthe computer.

In addition, the present invention incorporates a case wherein thesupplied program codes are stored in the memory of a function expansionboard in the computer or a function expansion unit connected to thecomputer, the CPU of the function expansion board or function expansionunit performs part or all of actual processing on the basis of theinstructions of the program codes, and the functions of the aboveembodiments are realized by the processing.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

1. An image capturing apparatus comprising: an image sensor thatincludes a plurality of pixels arranged in the form of a matrix, whereineach pixel in the plurality of pixels includes a photoelectricconversion unit, an amplifying unit that amplifies pixel signals outputfrom the photoelectric conversion units, and an addition unit that addsthe pixel signals of the same color amplified by the amplifying unit;and a control unit that controls the amplifying unit to amplify thepixel signals with amplification factors that are independently set foreach pixel, when the pixel signals of the same color are added by theaddition unit.
 2. The image capturing apparatus according to claim 1,wherein a number of pixel signals to be added by the addition unit isdifferent for each color.
 3. The image capturing apparatus according toclaim 1, wherein the control unit reads out the pixel signals bythinning the pixel signals output from the image sensor with apredetermined thinning factor.
 4. The image capturing apparatusaccording to claim 3, wherein the amplification factors areindependently set for each pixel in accordance with the thinning factor.